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O ! Abstract 

We enumerate the pattern class Av(2143, 4231) and completely describe its permutations. 
The main tools are simple permutations and monotone grid classes. 
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1 Introduction 



This paper is a contribution to an ongoing endeavour initiated by Knuth in section 2.2.1 of [10]: 
Q\ [ in Exercises 7-11 (and their solutions) he enumerated the permutations that can be obtained 

from an input-restricted deque by first proving that they are exactly those that do not contain 
either of the patterns 4213 and 4231. Since that time the subject of Permutation Patterns has 
developed into a rich combinatorial theory one of whose central concerns continues to be the 
description and enumeration of permutations that do not contain a stipulated set of patterns. 

For completeness we briefly recall the salient definitions. A permutation is simply an ar- 
rangement of the numbers 1, 2, ... n for some n > (note therefore that all our permutations 
will be non-empty). A permutation n is said to be contained in (or be a subpermutation of) an- 
other permutation a if a has a subsequence whose terms are ordered in the same relative way as 
those of tc. For example, 3142 is contained in 1573462 because the subsequence 5362 is ordered 
in the same way as 3142. If n is not contained in a we say that a avoids n. The subpermutation 
relation is obviously a partial order on the set of all permutations and its down-sets are called 
pattern classes. For every pattern class V there is a (possibly infinite) set of permutations that 
do not belong to V and are minimal with respect to not lying in V . This set, B say, is called the 
basis of V and it determines V as exactly that set of permutations that avoid every member of 
B: we write V = Av(B). 

The subpermutation order is invariant under the 8 symmetries generated by inversion, re- 
versal and complementation. These symmetries often allow arguments by case enumeration to 
be condensed. 
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If the set B contains any permutation of length 1 or 2 then it is trivial to identify Av(£>) and 
to enumerate it. If B contains 132 (or any of its 3 symmetries 213, 231, or 312) then Av(B) can 
be enumerated by the methods of [1]. If B is only known to contain 123 (or its symmetry 321) 
much less has been proved although many special cases have been solved (see, for example, 
[3, 7, 15, 16]). 

Knuth's problem above was the first one to be solved with B having two permutations both 
of length 4. There are in fact 56 essentially different such problems (i.e. problems in which 
the sets B are inequivalent under symmetries). It is known that they give rise to 38 different 
enumerations [8, 11, 12, 13, 14] (some inequivalent pairs are Wilf-equivalent meaning that they 
nevertheless have the same enumeration). Of these 38 Wilf classes around half have yet to be 
enumerated (see [18] which lists 18 published enumerations). 

The ones that have been enumerated have been natural testing grounds for a succession 
of techniques (such as generating trees [17], the insertion encoding [4], the Schensted corre- 
spondence [5], and simple permutations [1]). In this paper we apply a combination of recently 
devised techniques to the pattern class Av(2143,4231) and, not only do we enumerate it, we 
give a complete structural description of its members. 

Our main technical tools are simple permutations and their inflations, together with con- 
strained decompositions of permutations into monotone subsequences and we now summarise 
the basic facts we shall need about these. 

A simple permutation is one with no non-trivial intervals. In this context an interval of a 
permutation is just a contiguous subsequence whose values form a consecutive set of inte- 
gers. For example 546 is an interval of 3154627. If the interval is either a singleton or the 
entire permutation then it is trivial. Simple permutations are precisely those that do not arise 
from a non-trivial inflation, in the following sense. Let a be any permutation of length m and 
ct\, et2,...,0Cm an y sequence of permutations. Then the inflation of a by ot\, oli,..., ot m , which we 
denote by o~\cl\,k.2,- ■ - ^m], is that permutation of length |«i| + • • ■ + \ot m \ which decomposes 
into m segments oc^a^ ■ ■ ■ oc' n where each segment a- is an interval that is order isomorphic to 
&i, and the sequence a\a^ - ■ ■ a n formed by any (and hence every) choice of a\ from is order 
isomorphic to a. For example the inflation of 3142 by 21, 132, 1, 123 is 

3142 [21, 132, 1, 123] = 87 132 9 456 

Permutations that are inflations of 12 and 21, which occur often in this paper, are said to be, 
respectively, sum decomposable and skew decomposable. The precise connection between simple 
permutations and inflations is furnished by a result from [1]. 

Proposition 1.1. Let n be any permutation. Then there is a unique simple permutation a and permu- 
tations 0L\,...,0L n such that 

n = a[a 1 ,...,a n ]. 

If a 7^ 12, 21, then 0L\,...,CL n are also uniquely determined by n. If n = 12 or 21, then oc\, &2 are unique 
so long as we require that «i is sum indecomposable or skew indecomposable respectively. 

Our other technical tool is a diagrammatic expositional aid. We regard a permutation n as a 
set of points [i, n(i)) lying in an n x n grid within the plane. We partition such a square grid into 
cells using a fixed number of vertical and horizontal dividing lines. The points that lie within a 
cell then define a subsequence of n and we shall be particularly interested in when these sub- 
sequences are monotone. Consider, for example, the permutation [6, 12, 11, 7, 10, 4, 5, 9, 3, 8, 2, 1] . 
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Figure 1: The permutation [6, 12, 11, 7, 10, 4, 5, 9, 3, 8, 2, 1] on a 3 x 2 grid 

As shown in Figure 1 we can represent it on a 3 x 2 grid with one empty cell, two increasing 
cells, and three decreasing cells. 

The set of permutations whose diagram can be divided into cells with a fixed number of 
vertical and horizontal lines, where it is stipulated that each cell should contain either monotone 
increasing points, monotone decreasing points, a single point, or be empty, defines a pattern 
class, called a grid class. We may denote such a grid class by specifying its matrix of cells 
with each cell containing +1 for increasing, —1 for decreasing, a dot for a single point and 
for empty. It is convenient sometimes to denote the content of an increasing (respectively 
decreasing) cell by a line ascending (respectively descending) to the right. So the permutation 
[6, 12, 11, 7, 10, 4, 5, 9, 3, 8, 2, 1] belongs to the grid class whose matrix is 




or whose pictorial representation is 




The bulk of our paper is an analysis of Av(2143,4231). We determine its simple permuta- 
tions and describe them in grid class terminology. Next we examine how the simple permu- 
tations can be inflated and thereby we obtain a complete description of the pattern class. The 
enumeration calculation is then carried out using encodings of permutations derived from the 
grid class description. The paper ends with some remarks on pattern classes for which similar 
analyses may be possible. 



2 The structure of Av(2143, 4231) 
2.1 The simple permutations 

We shall obtain the general form of simple permutations in the class by a division into cases 
according to the pattern determined by the four extremal points of the permutation. Let these 
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Figure 2: The extremal points form the pattern 3412 
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Figure 3: The extremal points and an interior point form the pattern 42513. 

points be denoted by £ the leftmost point, r the rightmost, u the highest and d the lowest. In 
a simple permutation of length at least 4 these points are all distinct and their pattern will 
therefore be one of 2143 (which is impossible as it is a basis element of the class), 3412, 2413 
and 3142. 

Lemma 2.1. If n is a simple permutation in Av(2143,4231) and the pattern determined by £, r, u and 
d is 3412 then n is one of 42513 or 35142. 

Proof. Consider the regions of n depicted in Figure 2. By the 2143 avoidance one of the regions 
BUC and D U G is empty. 

To within a reverse complement symmetry we may take B U C to be empty. But then region 
G is not empty. If it were then, because lAu is not an interval, D is not empty; however all 
points of region E U F lie below all points of region D as 4231 is not a subpermutation of n; and 
now it would follow that the points of A U D together with I and u would be an interval. 

Thus G contains some point g and the four extremal points of a together with g give rise 
to a 42513 pattern. We are now in the situation depicted in Figure 3. Of the 16 square regions, 
the 10 unlabeled regions must be empty in order to avoid either 4231 or 2143. We claim that 
in order for a to be simple, the remaining six labeled regions must be empty. To justify this 
claim we begin by noting that every point of P must lie below every point in Q, as otherwise 
we would find a 4231-pattern, using the points d and g. Thus the points in region Q, together 
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Figure 4: Simple permutations where the extremal points form the pattern 3142 



with u, form an interval, and since cr is simple we infer that Q must be empty. Now, however, 
the two regions P and R, together with the point £ form an interval, from which we conclude 
that both P and R must be empty. The regions S, T and U must also be empty by a similar 
argument (or by taking inverses). Hence a = 42513. Similarly, when D U G is empty, we have 
cr = 35142. □ 

Lemma 2.2. If n is a simple permutation in Av(2143,4231) and the pattern determined by £, r, u and 
d is 3142 then n has the form shown in Figure 4. In this figure the letters A,B,C,D,E label the cells 
that contain them. Unlabelled cells are empty. Furthermore cells C and E are empty and possibly cells A 
and G are empty also. 



Proof. The initial situation is depicted in Figure 5 where the two unlabeled regions are empty 
since n avoids 2143. The rectangle comprising cells C, D, E must form a decreasing pattern to 
avoid 4231: in particular, this means that points in the cell labeled C lie above all points in 
cells D and E. Thus, to avoid a non-trivial interval being formed by the point £ together with 
the points in cells A and C, there must exist some point a G A lying above some point b G B. 
However, any point in C to the left of a now participates as the '1' in a 2143 with £, a and b, 
while any point in C to the right of a forms the '2' of a 4231 with a, b and r. Thus C must be 
empty. 

Because of the 2143-avoidance B is increasing. It cannot be empty except in a trivial case 
where A, which would now form an interval with £, is also empty; so we can let b denote the 
lowest point of B. The points of A must all lie above b; for, if A contained points below b, then 
these points together with £ would have to be separated by a point of A that was greater than 
b and then there would be a 4231 pattern. Furthermore no point of A can lie above the second 
largest point of B (if B has such a point) because, again, a 4231 pattern would be created. But 
now the points of A form an interval and so A has at most one point ( which, when it exists, 
lies in value between the lowest and second lowest points of B). 

A symmetric argument applies to the regions E (which must be empty), F (which contains 
an increasing sequence) and G (which contains at most one point, below exactly one point of 
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Figure 5: The extremal points form the pattern 3142 

F), and from this we conclude that the permutation has the form given in Figure 4. □ 

In the remaining case that the pattern of the extremal points is 2413 (which is the inverse of 
3142) we have 

Corollary 2.3. If re is a simple permutation in Av(2143,4231) and the pattern determined by £, r, u 
and d is 2413 then n^ 1 has the form shown in Figure 4. 

2.2 Inflating the simple permutations 

Now we determine which inflations of simple permutations lie in Av(2143, 4231) and we start 
with the simple permutations of Lemma 2.2. Let D be the grid class whose matrix is the column 

( M 

vector — 1 . Then, by Lemma 2.2, a simple permutation a whose extremal points have the 
^ 1 / 

pattern 3142 has one of four types: 

• a G T> — we will call these permutations type 1 simple permutations, 

• a g V, but a - cr(l) eP- type 2, 

• a ^T>, but a — cr(n) EV — type 3, or 

• a ^T> and not of types 2 or 3, but a — tr(l) — o~(n) G V — type 4. 

These types correspond, respectively, to cells A and G (as defined in Lemma 2.2) having 
sizes (0, 0), (1, 0), (0, 1) and (1, 1). The cell B together with u, the cell D, and the cell F together 
with d (also defined in Lemma 2.2) are the three cells of V, read from top to bottom. Note that 
types 2 and 3 can be obtained from each other by the reverse complement symmetry, so we will 
handle these two cases together. We will let £ = Av(2143,312) and T = Av(2143,231); these 
classes have grid class descriptions as depicted in Figure 6. 

Lemma 2.4. A type 1 simple permutation can be inflated as follows: 

• The first point can be inflated by £. 

• The last point can be inflated by T. 

• The points of regions B and F and the points u and d can be inflated by Av(21). 
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Figure 6: The pattern classes £ and T 

• The points of region D can be inflated by Av(12). 

Proof. Let a be a type 1 simple permutation in Av(2143 / 4231) of length n. The first and last 
points can be inflated by £ and T respectively: for example, since the bottom element of cr lies 
to the right of the leftmost point, inflating the leftmost point by 312 would give rise to a 4231 
pattern. Thus the leftmost point must avoid 312, in addition to 2143, but there are no further 
restrictions. 

Every point in the increasing region F can be inflated only by permutations from Av(21): to 
see this, note that the last two points of a form a 21-pattern above and to the right of all points 
in this region, so inflations of points of F must avoid 21 in order not to create a 2143 pattern. 
A similar argument applies to the points in the increasing region B. For the interior points in 
the decreasing region D, note that each forms the '2' of a 321-pattern with the leftmost and 
rightmost points of a. Thus points of D can only be inflated by Av(12) in order not to create a 
4231-pattern. □ 

The same type of analysis proves: 

Lemma 2.5. A type 2 simple permutation of length n can be inflated as follows: 

• The second point (the single point of region A) can be inflated by £. 

• The last point can be inflated by T. 

• The first point, the points d and u, all points in region B apart from its first, and all points in 
region F can be inflated by Av(21). 

• The points of region D can be inflated by Av(12). 

• The first point of region B cannot be inflated. 

An analogous argument holds by considering the reverse complement symmetry for type 3 
simple permutations. For type 4 permutations, a similar argument yields: 

Lemma 2.6. A type 4 simple permutation of length n can be inflated as follows: 

• The second point can be inflated by £. 

• The penultimate point can be inflated by T. 

• The first point, the last point, all points in region B apart from its first, and all points in region F 
apart from its last can be inflated by Av(21). 
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• The points of region D can be inflated by Av(12). 



• The first point of region B and the last point of region F cannot be inflated. 

Now that we know the inflations of the simple permutations covered by Lemma 2.2 we can 
obtain analogous results for the simple permutations of Corollary 2.3 by taking inverses. 

Finally we observe that the two sporadic permutations 42513 and 35142 behave as type 2 or 
type 3 permutations, and their inflations are the same. Indeed, 35142 is visibly a subpermuta- 
tion of any of the permutations depicted in Figure 4 and so its inflations are subpermutations 
of their inflations. A similar property holds (by the reverse complement symmetry) for the 
inflations of 42513. 

It follows from these proofs and the preceding remarks that the class Av(2143, 4231) is equal 
to the union of the 2 grid classes represented by the picture: 
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3 The enumeration of Av(2143,4231) 

We are now able to compute the generating function of the class by using Proposition 1.1. This 
proposition tells us that the class is the disjoint union of the permutation 1, its set of sum 
decomposable permutations, its skew decomposable permutations, and its inflations of simple 
permutations of length at least 4. Its generating function is therefore the sum of the generating 
functions of these subsets. 

To help in the computation we introduce generating functions for some auxiliary subclasses 
of Av(2143, 4231). Let d{x) = x/ {1 — x) be the generating function for the set of decreasing (or 
increasing) permutations. Furthermore let e(x) = *}^ x + x i be the generating function for the set 
£ = Av(312,2143); it is also the generating function for the set T = Av(231,2143) (see e.g. [6]). 

We will begin by computing fe(x), the generating function of the skew decomposable per- 
mutations of Av (2143, 4231). First note that if we write a skew decomposable permutation re 
as n = tz\ O n i, then n\ € £ and Ti2 G J 7 (whose generating functions are both e(x)). To make 
this decomposition unique, we insist that 774 is skew indecomposable, so we briefly turn our at- 
tention to the skew indecomposable permutations in Av(312, 2143), whose generating function 
we will denote by e^x). The generating function for the skew decomposable permutations in 
Av(2143,4231) will then be given by f e (x) = e (»e(x). 

Let t be an skew indecomposable permutation in £ = Av(312, 2143). As this class contains 
no simple permutations of length 4 or more, T is either the permutation 1 or is sum decom- 
posable. In the latter case, write t = T\ © T2 where Ti is sum indecomposable. If X\ = 1 then 
T2 has no restrictions other than avoiding 312 and 2143, while if T\ contains 21 then T2 must be 
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increasing. Recalling that the generating function for the class is e(x), the skew indecomposable 
permutations satisfy e$(x) = x + xe(x) + (e(x) — e0(x))d(x). Solving, yields: 

x(l-x) 2 



l-3x + x 2 

and so the generating function for the skew decomposables in Av (2143, 4231) is 

/eW= (1 _ 3x + x2)2 . 

We now turn our attention to the sum decomposable permutations. Consider n = n\ © Ti2, 
written so that ii\ is sum indecomposable. If ii\ = 1, then TI2 can be any element of the 
class Av(2143,4231). Otherwise, n\ must contain 21, so TC2 must be increasing to avoid 2143. 
Thus the generating function /® (x) for the sum decomposable permutations satisfies /© (x) = 
x/(x) + (/(x)-/ e (x)-x)d(x). 

It remains to consider the inflations of the simple permutations of length 4 or more. As 
in the previous section we divide the analysis according to the pattern determined by the 4 
extremal points and we begin with the case that this pattern is 3142. We follow our previous 
analysis of permutation types by defining the generating functions for the simple permutations 
of types 1 — 4 to be S\(x) , si{x) , s$(x) and s±(x) respectively. Note that sj{x) = S3(x). 

Reading from left to right in the permutation, we encode points of a simple permutation 
lying in the grid class V using the three letters a, b and c: a represents a point in the lowest 
cell, b a point in the middle and c a point in the top. To enforce uniqueness, we insist that 
if a point can be encoded by b then it should be. For example, the encoding of 51647283 is 
bacbcacb. For the permutation to be simple, the word must not contain any factors aa, bb or 
cc (as otherwise these two points will form an interval of size 2), and additionally for type 1 
simple permutations the encoding must start with ba and end with cb. (Note that for types 2 
and 3, we will drop one of these end conditions, while for type 4 we drop both end conditions.) 

Type 1 enumeration. For n > 4, these permutations are in bijection with words of length n 
over {a, b, c} of the form ba... cb, and with no factor aa, bb or cc. There is one word of length 
4, namely bacb, and one of length 5, babcb. Consider a word w = W\ . . . w n of length n > 6. If 
the fourth symbol from the right, w n -3 = a or b, then the word W\ . . . Vd n -sW n -\W n is a valid 
word of length n — 1. On the other hand, if iy„_3 = c, then the word W\ . . . w n -i±w n -\"w n is a 
valid word of length n — 2, where the other omitted symbol iy„_2 could have been either a or b. 
Hence we obtain the recurrence S\ in = Si /W _i + 2si „_ 2 / where Si ; „ denotes the coefficient of x" 
in Si(x). Solving, yields the generating function 



si(x) = 



X 4 



(l-2x)(l + x)' 



Thus the generating function for the number of permutations in Av (2143, 4231) that are infla- 
tions of type 1 simple permutations is 

fl (x) = Sl (d(x)) ■ ^ = {1 _ 3x){1 _ 3x + x2)2 - 
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Type 2 enumeration. Here, we require the four leftmost points of a to take the form of the 
four leftmost points as depicted in Figure 4. The remainder of the permutation lies in the grid 
class T>. Thus we enumerate a permutation of length n by considering words of length n — 4 
over a, b, c. As before these have no repeated letters as factors, but note that we can now drop 
the condition that the word starts with ba as the four points placed at the left of the permutation 
guarantee that no interval can be found here. Thus, for n > 6 the number b„ of type 2 simple 
permutations of length n is given by the number of words of length n — 4 of the form . . . cb 
with no repeated letters as factors. Reading the word from right to left, this is easily seen to 
give 2"~ 6 choices for n > 6, giving the generating function S2(x) = ^ 2x ) • Consequently, the 
generating functions for permutations that are inflations of type 2 simple permutations is 

x • cixS 2 x^ 

f 2 (x) = S 2 (d(x)) ■ = {1 _ 3x)[1 _ 3x + x2r 

Type 3 enumeration. Using the reverse complement symmetry, we immediately obtain s^(x) = 
s 2 (x) and / 3 (x) = fi(x). 

Type 4 enumeration. Here, the first four and last four points of the permutation must be fixed 
as shown in Figure 4. We encode the intermediate points as before, but note now that we have 
no end restrictions at either end. Thus there is one of length 8, while for n > 9 there are 3 • 2™~ 9 
possible choices, and hence we obtain the generating function s±(x) = 12t . The generating 
function for the inflations of these simple permutations is then 

x 2 ■ e(x) 2 x 8 



/ 4 (x) = s 4 (d(x)) 



d(x) 4 (1 -3x)(l -x) 2 (l -3x + x 2 ) 2 ' 



Because of the inversion symmetry we obtain exactly the same generating functions when 
the 4 extremal points have the pattern 2413. 

To complete the enumeration we have to consider the permutations 42513 and 35142 (when 
the 4 extremal points have the pattern 3412). They behave like types 2 and 3 respectively and 
so we find the generating function for inflations of these two sporadic simple permutations to 
be2.x.e(x) 2 .d(x) 2 = JT ^ 7 . 

This analysis into cases is obviously without overlaps. Thus the generating function s(x) for 
the simple permutations of length 4 or more in Av(2143,4231) is s(x) = 2x 5 + 2(si(x) + S2(x) + 
S3(x) + S4(x)) and so is given by: 

_ 2x 4 (l + x + x 2 + x 4 + 2x 5 + x 6 ) 
S(X) " (l-2x)(l + x) 

The first few terms of the sequence (starting at n = 4) are 2, 4, 10, 18, 40, 80, 162. 
Likewise, the generating function for the entire class Av(2143, 4231) is: 

2x 5 

f(x) =* + /©(*) +/ e (x) + {1 _ 3x + x2)2 + 2 (/i(*)+/2(*)+/3(*) +Mx)) 

We have explicit formulae for every term on the right-hand side of this equation except for the 
term /©(x). However, for this term we have an equation that relates it to f(x) itself and solving 
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the resulting equations gives 

x - llx 2 + 51x 3 - 127x 4 + 186x 5 - 165x 6 + 87x 7 - 23x 8 + 3x 9 

4 Some related pattern classes 

The pattern class we have considered in this paper is the first class of the form Av(a, /3) with 
l a l = = ^ that has been analysed by means of grid classes. We expect that grid classes 
will play an increasingly important role in the study of pattern classes. A forthcoming paper 
[2] gives some very general conditions under which a grid class can be defined by finitely 
many forbidden permutations, has a rational generating function, and is partially well-ordered. 
Furthermore there is already a useful criterion [9] for a given pattern class to be contained 
within a grid class. For example, this criterion applies to Av(2143,4321), Av(2143,4312) and 
Av(1324, 4312), all hitherto unenumerated. We expect to report on these pattern classes in a 
future paper. 
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